Mobile terminal security systems

ABSTRACT

Systems that can track an object as it moves across the field of view of the security device, integrate transaction data with the object information, and can identify events that trigger an alert and methods for using the systems.

BACKGROUND

This invention relates generally to security systems and methods forusing security systems, and, more particularly, to security systemstracking objects across cameras in environments including mobile POScomponents.

A number of systems, using a number of cameras, enable identifying anobject and following the object as it moves from one camera to anothercamera. Such systems find applications in airport security and similarsituations.

In retail environments, the point of sale (POS) area is typicallystrongly monitored and movement in and out of that area is tracked.Recent developments have given rise to Mobile POS components wheretransactions occurred at any point in the environment A customer canmake multiple mobile PUS transactions and stop at multiple points in theenvironment. The mobile environment can give rise to situations wherealert should be triggered.

The other similar environments having the same needs.

There is a need for systems that can track an object as it moves acrossthe field of view of the security device, integrate transaction datawith the object information, and can identify events that trigger analert.

BRIEF SUMMARY

Systems that can track an object as it moves across the field of view ofthe security device, integrate transaction data with the objectinformation, and can identify events that trigger an alert and methodsfor using the systems are disclosed herein below.

In one or more embodiments, the method of these teachings includesacquiring and recording an image of an object from one camera of anumber of security cameras; each camera from the number of securitycameras having position information and calibrated to identify othercameras from the number of security cameras in the vicinity of the eachcamera, providing the object with an identifier if the object meetspredetermined criteria; the identifier provided to the object being acharacteristic of the object, linking the object with transaction datafrom a mobile transaction component (also referred to as a mobile POScomponent), tracking the object from the one camera to another camera inthe vicinity of the one camera as the object moves, providing video datafor the object and the transaction data link to the object to a server;the video data for the object and the transaction data link to theobject being stored at a database, utilizing the server to determinewhether an event requiring an alert has occurred and utilizing theserver to manage alerts.

In one instance, the predetermined criteria includes completing atransaction at the mobile POS component.

In one or more instances, the label is used in tracking the object fromthe one camera to the other camera; the label and the objectconstituting a labeled object.

In one or more embodiments, the system of these teachings includes anumber of security cameras; each camera from the number of securitycameras having position information and calibrated to identify othercameras from the number of security cameras in the vicinity of eachcamera; an image of an object being acquired and recorded by onesecurity camera from the number of security cameras, a server, adatabase, the database having information for logic methods, one or moreprocessors, and one or more computer usable media having computerreadable code embodied therein; the computer readable code, whenexecuted in the one or more processors, causes the one or moreprocessors to receive the image of the object, provide the object withan identifier if the object meets predetermined criteria; the identifierprovided to the object being a characteristic of the object, link theobject with transaction data from a mobile POS component, and providevideo data for the object and the transaction data link to the object tothe server. This server includes one or more other processors and one ormore other computer usable medium that has computer readable codeembodied therein, which, when executed by the one or more otherprocessors, causes the one or more other processors to receive the videodata for the object and the transaction data link to the object, storethe video data for the object and the transaction data link to theobject in the database, determine, using the logic methods, whether anevent requiring an alert has occurred and manage alerts.

For a better understanding of the present teachings, together with otherand further objects thereof, reference is made to the accompanyingdrawings and detailed description and its scope will be pointed out inthe appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic representation of components of one embodiment ofthe system of these teachings;

FIG. 2 is a chart that shows the key components and their relationshipwith other components in the present teachings;

FIG. 3 is a chart that shows the device interaction between thecomponents of FIG. 1 (also referred to as cameras or alarm system) andthe server in the present teachings;

FIG. 4 represents a chart of the sub-components contained in the mobiletransaction component as set forth in the present teachings; and

FIG. 5 is a block diagram representation of a portion of the server (orremote console) of these teachings.

DETAILED DESCRIPTION

The following detailed description is of the best currently contemplatedmodes of carrying out these teachings. The description is not to betaken in a limiting sense, but is made merely for the purpose ofillustrating the general principles of these teachings, since the scopeof these teachings is best defined by the appended claims. Although theteachings have been described with respect to various embodiments, itshould be realized these teachings are also capable of a wide variety offurther and other embodiments within the spirit and scope of theappended claims.

As used herein, the singular forms “a,” “an,” and “the” include theplural reference unless the context clearly dictates otherwise.

Except where otherwise indicated, all numbers expressing quantities ofingredients, reaction conditions, and so forth used in the specificationand claims are to be understood as being modified in all instances bythe term “about.”

In one or more embodiments, the system of these teachings includes anumber of security cameras; each camera from the number of securitycameras having position information and calibrated to identify othercameras from the number of security cameras in the vicinity of eachcamera; an image of an object being acquired and recorded by onesecurity camera from the number of security cameras, a server, adatabase, the database having information for logic methods (computerreadable code for the logic methods being stored in the database), oneor more processors, and one or more computer usable media havingcomputer readable code embodied therein; the computer readable code,when executed in the one or more processors, causes the one or moreprocessors to receive the image of the object, provide the object withan identifier if the object meets predetermined criteria; the identifierprovided to the object being a characteristic of the object, link theobject with transaction data from a mobile transaction component (alsoreferred to as a mobile point-of-sale (POS) component), and providevideo data for the object and the transaction data link to the object tothe server. This server includes one or more other processors and one ormore other computer usable medium that has computer readable codeembodied therein, which, when executed by the one or more otherprocessors, causes the one or more other processors to receive the videodata for the object and the transaction data link to the object, storethe video data for the object and the transaction data link to theobject in the database, determine, using the logic methods, whether anevent requiring an alert has occurred and manage alerts.

In one instance, the identifier provides a label for the object, wherethe label is used in tracking the object from the one camera to theother camera. The label and a object constitute a labeled object. Inanother instance, the predetermined criteria includes completing atransaction at the mobile POS component.

In a further instance, when the field of view (FOV) of each one cameraoverlaps the field of view of one or more other cameras, after theobject is initially acquired, labeled and tracked by one of the cameras,tracking of the labeled object is handed from the one camera to theother, when the labeled object moves into the field of view of the othercamera.

In one or more embodiments, in managing alerts, the computer readablecode in the other computer usable medium at the server, causes, whenexecuted in the one or more other processors, the one or more otherprocessors to determine, using the logic methods stored in the database,whether an alert has occurred, determine, using the logic methods, atype of alert, merge an identifying ID with the alert type, determininga location of the alert and transmit the merged ID and the alert typeand location to a predetermined site.

A component of one embodiment of the system of the present teachings isshown in FIG. 1. Referring to FIG. 1, in the embodiment shown therein,an object (a customer) 35 is moving along two structures 25 (such asdisplays, shelves, etc.) a number of cameras 15 are disposed above theobject 35 and the structures 25. Each camera has a field of view (FOV)20 in the embodiment shown there fields of view 20 are overlapping. Eachcamera has position information as to the position of the camera and iscalibrated to identify other cameras in the vicinity, in one instance,the calibration identifies the cameras in the vicinity that haveoverlapping fields of view. A mobile transaction terminal 30 is disposedin the aisle between the two structures 25.

In one exemplary embodiment, typical of many retail environments, thetwo structures are shelves containing merchandise, the mobiletransaction terminal 30 is a mobile POS terminal and the object 35 is aconsumer. As a consumer moves along the aisle, an image of the consumer35 is acquired by at least one of the cameras 15. As the consumer stops,makes a transaction (such as a purchase) at the mobile PUS terminal 30,the image of the consumer is provided with an identifier (label) whichbecomes a characteristic of the image of that object (consumer). As aconsumer moves from the field of view of one camera to the field of viewof another camera, the image of the consumer is tracked and the image ofthe object is linked with transaction data from the mobile POS terminal30. Video data for the consumer and the transaction data link to theconsumer is provided to a server. At this server, it is determiningwhether the event, the trajectory of the consumer after the transactionat the mobile POS term, requires an alert. An Exemplary embodiment of apossible situation that may require an alert is a consumer stopping atlocations with similar products as that involved in the mobile POStransaction and selecting one similar product.

FIG. 2 shows an embodiment of the system of the present teachings,Referring to FIG. 2, in the embodiment shown therein, the componentsshown in FIG. 1, referred to as image acquisition devices (cameras), 110provide data through a communication network 130, labeled as an Internet(although any communication network including local area communicationnetworks could be used), to a server 140. In the embodiment shown inFIG. 1, a device Gateway 120 and a remote console 125 are shown asoptional (dotted) components. In one instance, some of the functionscould be performed at the remote console 125 while, in another instance,all of the functions can be performed at the server 140. The term imageacquisition devices, as used above, includes the mobile transactionterminal. The mobile terminal aspects of the system of these teachingsare included in the component labeled image acquisition devices 110 andin the application that the server is adapted to perform (being adaptedby having computer usable media having computer readable code embodiedtherein that when executed by a processors causes the server to performthe functions). (Without the mobile terminal aspects, the system of FIG.2 has the same appearance as a Web based security system such as thatdisclosed in U.S. Pat. No. 8,508,355, which is incorporated by referenceherein in its entirety and for all purposes.)

In one or more embodiments, the method of these teachings includesacquiring and recording an image of an object from one camera of anumber of security cameras; each camera from the number of securitycameras having position information and calibrated to identify othercameras from the number of security cameras in the vicinity of the eachcamera, providing the object with an identifier if the object meetspredetermined criteria; the identifier provided to the object being acharacteristic of the object, linking the object with transaction datafrom a mobile transaction component (also referred to as a mobile POScomponent), tracking the object from the one camera to another camera inthe vicinity of the one camera as the object moves, providing video datafor the object and the transaction data link to the object to a server;the video data for the object and the transaction data link to theobject being stored at a database, utilizing the server to determinewhether an event requiring an alert has occurred and utilizing theserver to manage alerts.

In one instance, the predetermined criteria includes completing atransaction at the mobile POS component.

In another instance, the mobile POS component receive location data froma location providing system (for example, but not limited to, GPS). Thelocation data for the mobile POS component is also linked to the objectthrough linking to the image of the object.

In one or more instances, the label is used in tracking the object fromthe one camera to the other camera; the label and the objectconstituting a labeled object.

In a further instance, when the field of view (FOV) of each one cameraoverlaps the field of view of one or more other cameras, the field ofview of the one camera, tracking of the labeled object is handed fromthe one camera to the other, when the labeled object moves into thefield of view of the other camera.

In one instance, the server 140 (or the remote console 125) includes adetected region labeling component (or computer readable code toimplement that function). The detected region labeling component couldbe, for example, a component as described in Multiscale ConditionalRandom Fields for Image Labeling, Proceedings of the 2004 IEEE ComputerSociety Conference on Computer Vision and Pattern Recognition, or inU.S. Pat. No. 7,024,020 B2, Apparatus and method for generatingobject-labeled image in video sequence, both of which are incorporatedby reference herein in their entirety and for all purposes. After thedetected region has been labeled, the detected region can be trackedduring motion (see, for example, Real-Time Object Tracking andClassification Using a Static Camera, Proceedings of the IEEE ICRA 2009Workshop on People Detection and Tracking Kobe, Japan, May 2009, Section14.6, Motion Tracking, in chapter 14, Machine Vision Book available athttp://www.cse.usf.edu/˜rlk/MachineVisionBook/MachineVision.files/MachineVision_Chapter14.pdf, both of which are incorporated by reference herein in theirentirety and for all purposes). The detected region can also be trackedduring motion using the methods described in unman Kang, Isaac Cohen andGerard Medioni, Continuous Tracking Within and Across Camera Streams,Proceedings of the 2003 IEEE Computer Society Conference on ComputerVision and Pattern Recognition (CVPR '03) and in Isaac Cohen, YunqianMa, Ben Miller, Associating Moving Objects Across Non-overlappingCameras: A Query-by-Example Approach, 2008 IEEE Conference onTechnologies for Homeland Security, and in U.S. Pat. No. 6,690,374, allof which are incorporated by reference herein in their entirety and forall purposes.

FIG. 4 shows a block diagram of one exemplary embodiment of a mobiletransaction (POS) component. It should be noted that other embodimentsare also within the scope of these teachings. The mobile transaction(PUS) component shown in FIG. 4 includes an antenna 304 configured tosend receive signals over a network. The antenna 304 is operativelyconnected to a communication module 308 that provides an interfacebetween the antenna and the network and one or more processors 312. Theone or more processors 312 are configured to execute machine readablecode which is stored on computer readable media 316 (such as a memory,as shown), The one or more processors 312 are also configured to storecomputer readable code in the computer readable media 316. The mobiletransaction (POS) component shown in FIG. 4 also has a display 334configured to present information, such as portions of the one or morevideo data streams, to a user and a data acquisition device (such as ascanner and or camera) 336. The data acquisition device allows acquiringdata for a transaction. A user interface 330 is configured to receiveuser input and the input to the one or more processors 312. In oneinstance, the user interface 330 is configured as part of the display334, as, for example, in a touch screen display.

FIG. 3 outlines the various components and the corresponding pathwaysincorporated into the server 140. The server 140 provides for thedetermination of whether an alert is required, the determination thetype of alert and the managing of alerts. Upon receiving the video datafor the object (the image of the object) from the component in FIG. 1(referred to as alarm system) 110, the object is provided with anidentifier if the object meets predetermined criteria, is linked withthe transaction data from the mobile POS component, and is tracked fromone camera to another camera, functions which are performed by eitherthe server 140 or the remote console 125. The video data (including thelocation data of each camera) and transaction data linked to the objectare provided to the server 140 and whether the video data transactiondata represent an alarm is determined 205. The determination isperformed using the logic methods stored in the database 135.

If the signal represents an alarm alert (210, FIG. 3), there are threemajor steps: 1) determine the Alert Type (220, FIG. 3); 2) merge ID withAlert Type (230, FIG. 3); and 3) transmit the alert via an appropriateavenue to a Monitoring Site, a designated site or a combination of theabove, or all of them at once. (270, FIG. 3). The logic stored in thedatabase 135 is used to determine the type of alert, wherein the storedlogic selects a user designated reaction to the type of alert. In oneembodiment, once the alarm's nature is determined, it is then necessaryto merge 230 the industry standard protocol with a Location ID databaseto determine which system or site is receiving the alert, Once the typeof alert and the location of the alert are determined, it is thennecessary to alert the Monitoring Site 170 (or a designated site, or allof them at once) via the Internet 130. Many Monitoring Sites utilizetelephone lines to receive alerts and, as such, it might be necessary toutilize a Smart Phone Emulator 250 so the telephone signals can bepassed via the Internet and be properly received by the Monitoring Siteor other receiving site. If, however, the Monitoring Site (or otherreceiving site) can receive and interpret IP signals, the alert can alsobe transmitted directly 260 to the Monitoring Site 170 via the Internet.

In one exemplary embodiment of the stored logic, these teachings notbeing limited only to the exemplary embodiment, the user designatedreaction is to review purchases at exit and the logic receivesinformation on the location and time spent at location as the consumer'simage is tracked over a number of cameras and also receives informationon the transaction, and location of the transaction, at the mobile POSterminal, Based on the information the logic identifies whether an alerthas occurred or not.

In one embodiment, the server 140 of these teachings includes one ormore processors (360, FIG. 5), one or more communication devices forcommunicating over a network (385, FIG. 5), one or more computer usablemedia having computer readable code embodied therein causing the one ormore processors to perform the method of the present teachings (380,FIG. 5). In one instance, the server also includes another computerusable medium having the database 350 described hereinabove embodiedtherein, One embodiment of the general structure of the server 140 isshown in FIG. 5.

In embodiments in which the functionality is completely embodied in theserver 140, the computer readable code embodied in the one or morecomputer usable media (380, FIG. 5), when executed by the one or moreprocessors, causes the one or more processors to receive the image ofthe object, provide the object with an identifier if the object meetspredetermined criteria; the identifier provided to the object being acharacteristic of the object, link the object with transaction data froma mobile transaction component (also referred to as a mobilepoint-of-sale (POS) component), provide video data for the object andthe transaction data link, receive the video data for the object and thetransaction data link to the object, store the video data for the objectand the transaction data link to the object in the database, determine,using the logic methods, whether an event requiring an alert hasoccurred and manage alerts.

In embodiments in which the remote console 125 is used, the generalstructure of the remote console 125 is the structure presented in FIG. 5without the database 350. In those embodiments, the computer readablecode embodied in the one or more computer usable media in the remoteconsole 125, when executed by the one or more processors, causes the oneor more processors to receive the image of the object, provide theobject with an identifier if the object meets predetermined criteria;the identifier provided to the object being a characteristic of theobject, link the object with transaction data from a mobile transactioncomponent (also referred to as a mobile point-of-sale (POS) component),provide video data for the object and the transaction data link to theserver. In those embodiments, the computer readable code embodied in theone or more computer usable media of the server, when executed by theone or more processors, causes the one or more processors to receive thevideo data for the object and the transaction data link to the object,store the video data for the object and the transaction data link to theobject in the database, determine, using the logic methods, whether anevent requiring an alert has occurred and manage alerts.

For the purposes of describing and defining the present teachings, it isnoted that the term “substantially” is utilized herein to represent theinherent degree of uncertainty that may be attributed to anyquantitative comparison, value, measurement, or other representation.The term “substantially” is also utilized herein to represent the degreeby which a quantitative representation may vary from a stated referencewithout resulting in a change in the basic function of the subjectmatter at issue.

Elements and components described herein may he further divided intoadditional components or joined together to form fewer components forperforming the same functions.

Each computer program may be implemented in any programming language,such as assembly language, machine language, a high-level proceduralprogramming language, or an object-oriented programming language. Theprogramming language may be a compiled or interpreted programminglanguage.

Each computer program may be implemented in a computer program producttangibly embodied in a computer-readable storage device for execution bya computer processor. Method steps of the invention may be performed bya computer processor executing a program tangibly embodied on acomputer-readable medium to perform functions of the invention byoperating on input and generating output.

Common forms of computer-readable media include, for example, a floppydisk, a flexible disk, hard disk, magnetic tape, or any other magneticmedium, a CDROM, any other optical medium, any physical medium withpatterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, any othermemory chip or cartridge, all of which are non-transitory. As stated inthe USPTO 2005 Interim Guidelines for Examination of Patent Applicationsfor Patent Subject Matter 1300 Off. Gaz. Pat, Office 142 (Nov. 22,2005), “On the other hand, from a technological standpoint, a signalencoded with functional descriptive material is similar to acomputer-readable memory encoded with functional descriptive material,in that they both create a functional interrelationship with a computer.In other words, a computer is able to execute the encoded functions,regardless of whether the format is a disk or a signal.”Although theinvention has been described with respect to various embodiments, itshould be realized these teachings are also capable of a wide variety offurther and other embodiments within the spirit and scope of theappended claims.

What is claimed is:
 1. A method for operating a security system in anenvironment including mobile point-of-sale (POS) components, the methodcomprising: acquiring and recording an image of an object from onecamera of a number of security cameras; each camera from the number ofsecurity cameras having position information and calibrated to identifyother cameras from the number of security cameras in the vicinity ofsaid each camera; providing the object with an identifier if the objectmeets predetermined criteria; the identifier provided to the objectbeing a characteristic of the object; linking the object withtransaction data from a mobile POS component; tracking the object fromsaid one camera to another camera in the vicinity of said one camera asthe object moves; providing video data for the object and thetransaction data link to the object to a server; the video data for theobject and the transaction data link to the object being stored at adatabase; and utilizing the server to determine whether an eventrequiring an alert has occurred; and utilizing the server to managealerts.
 2. The method of claim 1 wherein said identifier provides alabel for the object; said label being used in tracking the object fromsaid one camera to said another camera; said label and said objectconstituting a labeled object.
 3. The method of claim 1 wherein thepredetermined criteria comprises completing a transaction at the mobilePOS component.
 4. The method of claim 1 wherein mobile POS componentsreceive location data from a location providing system; wherein locationdata for said mobile POS component is also linked to the object.
 5. Themethod of claim 2 wherein a field of view (FOV) of each one camera fromthe number of cameras overlaps the fields of view of at least one othersecurity camera from the number of security cameras; the field of viewof said one camera overlapping the field of view of said another camera;and wherein tracking of the labeled object is handed from said onecamera to said another camera when the label object moves into the fieldof view of said another camera.
 6. The method of claim 1 wherein thestep of utilizing the remote server to manage alerts comprises:determining a type of alert for an alert using stored logic methods;wherein the stored logic methods select a user designated reaction;merging an identifying ID with the alert type; determining a location ofthe alert; and transmitting, using a transmitter and a preselectedtransmission method, the merged ID and alert type and the location to apredetermined site,
 7. A system comprising: a number of securitycameras; each camera from the number of security cameras having positioninformation and calibrated to identify other cameras from the number ofsecurity cameras in the vicinity of said each camera; an image of anobject being acquired and recorded by one security camera from thenumber of security cameras; a server; one or more processors; one ormore computer usable media having computer readable code embodiedtherein; the computer readable code, when executed in the one or moreprocessors, causes the one or more processors to: receive the image ofthe object; provide the object with an identifier if the object meetspredetermined criteria; the identifier provided to the object being acharacteristic of the object; link the object with transaction data froma mobile POS component; track the object from said one camera to anothercamera in the vicinity of said one camera as the object moves; andprovide video data for the object and the transaction data link to theobject to the server; a database, said database having information forlogic methods; the server comprising: one or more other processors; andone or more other computer usable media having other computer readablecode embodied therein; the other computer readable code, when executedin the one or more other processors, causes the one or more otherprocessors to: receive the video data for the object and the transactiondata link to the object; store the video data for the object and thetransaction data link to the object in the database; determine, usingthe logic methods, whether an event requiring an alert has occurred; andmanage alerts,
 8. The system of claim 7 wherein said identifier providesa label for the object; said label being used in tracking the objectfrom said one camera to said another camera; said label and said objectconstituting a labeled object.
 9. The system of claim 7 wherein thepredetermined criteria comprises completing a transaction at the mobilePOS component.
 10. The system of claim 7 wherein mobile POS componentsreceive location data from a location providing system; wherein locationdata for said mobile POS component is also linked to the object; andwherein said location data is also received by the server.
 11. Thesystem of claim 7 wherein the predetermined criteria comprises havingperformed a transaction with the mobile POS component.
 12. The system ofclaim 8 wherein a field of view (FOV) of each one camera from the numberof security cameras overlaps the fields of view of at least one othercamera from the number of security cameras; the field of view of saidone camera overlapping the field of view of said another camera; andwherein tracking of the labeled object is handed from said one camera tosaid another camera when the label object moves into the field of viewof said another camera.
 13. The system of claim 7 wherein, in managingalerts, said other computer readable code, when executed in the one ormore other processors, causes the one or more other processors todetermine a type of alert for an alert using the logic methods; merge anidentifying ID with the alert type; determine a location of the alert;and transmit, using a transmitter and a preselected transmission method,the merged ID and alert type and the location to a predetermined site.14. The system of claim 7 wherein said one or more other processors aresaid one or more processors; and wherein said one or more other computerusable media are said one or more computer usable media.